TWISTED CONJUGACY SEPARABLE GROUPS 



ALEXANDER FEL'SHTYN AND EVGENIJ TROITSKY 

Abstract. Wc study the notion of twisted conjugacy separability (essentially intro- 
duced in our previous paper for a proof of twisted version of Burnside-Frobenius theorem) 
and some related properties. We give examples of groups with and without this property 
and study its behavior under some extensions. An affirmative answer to the twisted 
Dehn conjugacy problem for polycyclic-by-finite group is obtained. Some problems for 
the further study are indicated. 
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1. Introduction 

Definition 1.1. Let G be a countable discrete group and (p : G ^ G an endomorphism. 
Two elements x,x' & G are said to be (p-conjugate or twisted conjugate, iff there exists 
g E G with x' = gx(f){g~^). We shall write {x}^ for the (p-conjugacy or twisted conjugacy 
class of the element x E G. The number of </)-conjugacy classes is called the Reidemeister 
number of an endomorphism and is denoted by R{4>)- If (f) is the identity map then the 
0-conjugacy classes are the usual conjugacy classes in the group G. 

If G is a finite group, then the classical Burnside-Frobenius theorem (see e.g. [37], 
[25, p. 140]) says that the number of classes of irreducible representations is equal to the 
number of conjugacy classes of elements of G. Let G be the unitary dual of G, i.e. the 
set of equivalence classes of unitary irreducible representations of G. The attempts to 
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generalize this theorem to the case of non-identical automorphism and of non-finite group 
(i.e., to identify the Reidemeister number of and the number of fixed points of on an 
appropriate dual object of G, provided that one of these numbers is finite) were inspired 
by the dynamical questions and were the subject of a series of papers [7, 8, 6, 12, 13, 9, 11]. 

In the present paper we study the following property for a countable discrete group G 
and its automorphism 0: we say that the group is (j)-conjugacy separable if its Reidemeister 
classes can be distinguished by homomorphisms onto finite groups, and we say that it is 
twisted conjugacy separable if it is 0-conjugacy separable for any automorphism with 
i?(0) < oo {strongly twisted conjugacy separable, if we remove this finiteness restriction) 
(Definitions 3.2 and 3.5). This notion was used in [11] to prove the twisted Burnside- 
Frobenius theorem for polycyclic-by-finite groups with the finite-dimensional part of the 
unitary dual G as an appropriate dual object. Related questions were studied in [2], [29]. 

After some preliminary considerations we prove the main results of the paper, namely 

(1) Classes of twisted conjugacy separable groups: Polycyclic-by-finite groups 
are strongly twisted conjugacy separable groups (Theorem 4.2). 

(2) Twisted conjugacy separability respects some extensions: Suppose, there 
is an extension H —>■ G ^ G/H, where the group if is a characteristic twisted 
conjugacy separable group; G/H is finitely generated FC-group (i.e., a group 
with finite conjugacy classes). Then G is a twisted conjugacy separable group (a 
reformulation of Theorem 5.1). 

(3) Examples of groups, which are not twisted conjugacy separable: HNN, 
Ivanov and Osin groups (Section 6). 

(4) The affirmative answer to the twisted Dehn conjugacy problem for 
polycyclic-by-finite groups (Section 7). 

(5) Residually finite groups are twisted conjugacy separable, in particular 
twisted Burnside-Probenius theorem is true for them in the following 
formulation: Let G be a residually finite group and its automorphism with 
i?(0) < OO. Then -R(0) = Sf{(f>), where S'/(0) is the number of fixed points of 
(j) : Gf ^ Gf, (f){p) = p o 0, where Gf is the part of the unitary dual G, which is 
formed by the finite-dimensional representations (Section 9). 

A number of examples of groups and automorphisms with finite Reidemeister numbers 
was obtained and studied in [6, 16, 14, 13, 9]. 

The interest in twisted conjugacy relations has its origins, in particular, in the Nielsen- 
Reidemeister fixed point theory (see, e.g. [23, 6]), in Selberg theory (see, eg. [39, 1]), and 
Algebraic Geometry (see, e.g. [19]). The congruences give some necessary conditions for 
the realization problem for Reidemeister numbers in topological dynamics. 
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2. Preliminary considerations 

The following construction relates 0-conjugacy classes and some conjugacy classes of 
another group. It was obtained in topological context by Boju Jiang and Laixiang Sun 
in [24]. Consider the action of Z on G, i.e. a homomorphism Z Aut(G), n i— ^ 0". Let 
r be a corresponding semi-direct product F = G* x Z: 

(1) T:=<G,t\tgr' = (j){g)> 

in terms of generators and relations, where t is a generator of Z. The group G is a normal 
subgroup of r. As a set, F has the form 

(2) r = u„ezG ■ r, 

where G ■ is the coset by G containing t". 

Remark 2.1. Any usual conjugacy class of F is contained in some G ■ t"'. Indeed, 
gg'f'g-^ = gg' (p''{g-^)t'' and ig'tH'^ = <t){g')f'. 

Lemma 2.2. Two elements x,y of G are (p-conjugate iff xt and yt are conjugate in the 
usual sense in F. Therefore g ^ g -t is a bijection from the set of (p- conjugacy classes of 
G onto the set of conjugacy classes ofV contained in G ■ t. 

Proof. If X and y are 0-conjugate then there is a G G such that gx = y4>{g). This 
implies gx = ytgt^^ and therefore g{xt) = {yt)g so xt and yt are conjugate in the usual 
sense in F. Conversely, suppose xt and yt are conjugate in F. Then there is a gf^ G F 
with gf^xt = ytgt"^. From the relation txt~^ = (pix) we obtain g(f)^{x)t'^~^^ = y(j){g)t'^^^ 
and therefore g(p'^{x) = y4>{g). Hence, y and are 0-conjugate. Thus, y and x are 

^-conjugate, because x and are always 0-conjugate: = x~^x(j){x). □ 

3. Twisted conjugacy separability 
We would like to give a generalization of the following well known notion. 

Definition 3.1. A group G is conjugacy separable if any pair g, h of non-conjugate 
elements of G are non-conjugate in some finite quotient of G. 

It was proved that polycyclic-by-finite groups are conjugacy separated ([34, 15], see also 
[36, Ch. 4]). Also, residually finite recursively presented Burnside p-groups constructed 
by R. I. Grigorchuk [18] and by N. Gupta and S. Sidki [20] are shown to be conjugacy 
separable when p is an odd prime in [40]. 

We can introduce the following notion, which coincides with the previous definition in 
the case = Id. 

Definition 3.2. A group G is cp-conjugacy separable with respect to an automorphism 
: G ^ G if any pair g, h of non-0-conjugate elements of G are non-0-conjugate in some 
finite quotient of G respecting (p. 

This notion is closely related to the notion RP(0) introduced in [11]. 

Definition 3.3. We say that a group G has the property RP if for any automorphism 
(p with R{(p) < oo the characteristic functions / of Reidemeister classes (hence all 
0-central functions) are PERIODIC in the following sense. 

There exists a finite group i^, its automorphism (px, and epimorphism F : G ^ K 
such that 



ALEXANDER FEL'SHTYN AND EVGENIJ TROITSKY 

1^1) The diagram 

G—^G 




commutes. 

(2) / = F* fxi where fx is a characteristic function of a subset of K. 
If this property holds for a concrete automorphism 0, we will denote this by RP(0). 

One gets immediately the following statement. 

Theorem 3.4. Suppose, R{(f)) < oo. Then G is (p-conjugacy separable if and only if G is 
RP(0). 

Proof Indeed, let Fij : G Kij distinguish ith and jth 0-conjugacy classes, where Kij 
are finite groups, i,j = Let F : G ^ (BijKij, F{g) = Ylii j ^ijid) ^ t)e the 

diagonal mapping and K its image. Then F : G ^ K gives RP((;/)). 

The opposite implication is evident. □ 

Definition 3.5. A group G is twisted conjugacy separable if it is 0-conjugacy separable 
for any with R{4>) < oo. 

A group G is strongly twisted conjugacy separable if it is 0-conjugacy separable for any 

0. 

Prom Theorem 3.4 one immediately obtains 

Corollary 3.6. A group G is twisted conjugacy separable if and only if it is RP. 

Theorem 3.7. Let F : T ^ K be a morphism onto a finite group K which separates two 
conjugacy classes ofV in G -t. Then the restriction Fq '■= F\g '■ G Im(F|G') separates 
the corresponding [by the bijection from Lemma 2.2) (p-conjugacy classes in G. 

Proof. First of all let us remark that Kst^Fg) is 0-invariant. Indeed, suppose Fdg) = 
F{g) = e. Then 

FG{<Pig)) = F{<P{g)) = F{tgt~') = = e 

(the kernel of F is a normal subgroup). 

Let gt and gt be some representatives of the mentioned conjugacy classes. Then 

F{{ht'')gt{ht'')-^) ^ F{gt), V/i G G, n G Z, 

Fihf'gt) ^ F(gtht''), \/heG,neZ, 

F{h(i/\gY'+^) ^ F{g(t){h)t''+^), meG,neZ, 

F{hr{g)) + F{Mh)), WheG,neZ, 

in particular, F{hg(j){h-^)) ^ F(g) "rfh e G. □ 

Theorem 3.8. Let some class of conjugacy separable groups be closed under taking semidi- 
rect products by Z. Then this class consists of strongly twisted conjugacy separable groups. 

Proof. This follows immediately from Theorem 3.7 and Theorem 3.4. □ 
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4. First examples: polycyclic-by-finite groups 

As an application we obtain another proof of the main theorem for polycyclic-by-finite 
groups. 

Let G' = [G, G] be the commutator subgroup or derived group of G, i.e. the subgroup 
generated by commutators. G' is invariant under any homomorphism, in particular it is 
normal. It is the smallest normal subgroup of G with an abelian factor group. Denoting 
:= G, G'(^) := G', := (G^"-^))', n > 2, one obtains derived series of G: 

(3) G = D G' D G(2) D ■ • ■ D G(") D . . . 

If = e for some n, i.e. the series (3) stabilizes by trivial group, the group G is 

solvable; 

Definition 4.1. A solvable group with derived series with cyclic factors is called polycyclic 
group. 

Theorem 4.2. Any polycyclic-by-finite group is a strongly twisted conjugacy separable 
group. 

Proof. The class of polycyclic-by-finite groups is closed under taking semidirect products 
by Z. Indeed, let G be an polycyclic-by-finite group. Then there exists a characteristic 
(polycyclic) subgroup P of finite index in G. Hence, P x Z is a polycyclic normal group 
of G X Z of the same finite index. 

Polycyclic-by-finite groups are conjugacy separable ([34, 15], see also [36, Ch. 4]). It 
remains to apply Theorem 3.8. □ 



5. Twisted conjugacy separability and extensions 

It is known that conjugacy separability does not respect extensions. In particular, in 
[17] an example of a group G which is not conjugacy separable, but contains a subgroup 
H of index 2 which is conjugacy separable, is given. 

For twisted conjugacy separable groups the situation is much better under some finite- 
ness conditions. More precisely one has the following statement. 

Theorem 5.1. Suppose, there exists a commutative diagram 



(4) 




G/H 



G/H 



0, 



where H is a normal subgroup of a finitely generated group G. Suppose, R{(f)) < oo , G/H 
is a FC group, i.e., all conjugacy classes are finite, and H is a (p' -conjugacy separable 
group. Then G is a (p-conjugacy separable group. 

Another variant of finiteness is \G/H\ < oo (without the property R{4>) < oo). In the 
relation to the next theorem, note that examples of a conjugacy separable subgroup of 
index 2 in a conjugacy non-separable group are known [17]. The proof of the next theorem 
is related to [29, Prop. 3.6]. 
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Theorem 5.2. Let H be a characteristic strongly twisted conjugacy separable subgroup of 
G. Then G is a strongly twisted conjugacy separable group. 

Proof. Obviously G is strongly twisted conjugacy separable if and only if {g}^ is closed 
in the profinite topology, where : G — G is an arbitrary automorphism and g & G is 
an arbitrary element. 

Suppose, coset representatives, where r is the index of H in G, and 

gi := Xig(j){x:r^). Then 

{g}^ = {wg(P{w-') \weG} = [j {hx,g4>{xr^)<P{h-') \ h e H} 

i 

= U \heH} = [j {{hg,^{h-')gr^)g, \ h e H} 



U i'U, 



■gi- 



Cosets by a normal subgroup of finite index are evidently closed and open in the profinite 
topology. Thus, by the supposition {ej^-g o0' is closed in H and G, as well as their right 
translations, i.e., entries of the above union. Since the union is finite, {g}<j) is closed in 
G. □ 



6. Examples and counterexamples 

Some of examples of groups, for which the twisted Burnside-Frobenius theorem in the 
above formulation is true, out of the class of polycyclic-by-finite groups were obtained by 
F. Indukaev [22]. Namely, it is proved that wreath products A I Z are RP groups, where 
^ is a finitely generated abelian group (these groups are residually finite). 

Now let us present some counterexamples to the twisted Burnside-Frobenius theorem 
in the above formulation for some discrete groups with extreme properties. Suppose, 
an infinite discrete group G has a finite number of conjugacy classes. Such examples 
can be found in [38] (HNN-group), [32, p. 471] (Ivanov group), and [33] (Osin group). 
Then evidently, the characteristic function of the unity element is not almost-periodic 
and the argument above is not valid. Moreover, let us show, that these groups give rise 
counterexamples to the above theorem. 

In particular, they are not twisted conjugacy separable. Evidently, they are not conju- 
gacy separable, because they are not residually finite. 

Example 6.1. For the Osin group the Reidemeister number R{ld) = 2, while there is only 
trivial (1-dimensional) finite-dimensional representation. Indeed, Osin group is an infinite 
finitely generated group G with exactly two conjugacy classes. All nontrivial elements 
of this group G are conjugate. So, the group G is simple, i.e. G has no nontrivial 
normal subgroup. This implies that group G is not residually finite (by definition of 
residually finite group). Hence, it is not linear (by Mal'cev theorem [27], [35, 15.1.6]) and 
has no finite-dimensional irreducible unitary representations with trivial kernel. Hence, 
by simplicity of G, it has no finite-dimensional irreducible unitary representation with 
nontrivial kernel, except of the trivial one. 

Let us remark that Osin group is non-amenable, contains the free group in two gener- 
ators F2, and has exponential growth. 
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Example 6.2. For large enough prime numbers p, the first examples of finitely generated 
infinite periodic groups with exactly p conjugacy classes were constructed by Ivanov as 
limits of hyperbolic groups (although hyperbolicity was not used explicitly) (see [32, 
Theorem 41.2]). Ivanov group G is infinite periodic 2-generator group, in contrast to the 
Osin group, which is torsion free. The Ivanov group G is also a simple group. The proof 
(kindly explained to us by M. Sapir) is the following. Denote by a and b the generators 
of G described in [32, Theorem 41.2]. In the proof of Theorem 41.2 on [32] it was shown 
that each of elements of G is conjugate in G to a power of generator a of order s. Let us 
consider any normal subgroup of G. Suppose 7 G iV. Then 7 = ga'^g~^ for some g E G 
and some s. Hence, = g~^'yg G and from periodicity of a, it follows that also a E N 
as well as a'^ G for any k, because p is prime. Then any element h oi G also belongs to 
being of the form h = ha'^{h)~^, for some k, i.e., N = G. Thus, the group G is simple. 
The discussion can be completed in the same way as in the case of Osin group. 

Example 6.3. In paper [21], Theorem III and its corollary, G. Higman, B. H. Neumann, 
and H. Neumann proved that any locally infinite countable group G can be embedded 
into a countable group G* in which all elements except the unit element are conjugate to 
each other (see also [38]). The discussion above related Osin group remains valid for G* 
groups. 

7. Twisted Dehn conjugacy problem 

The subject is closely related to some decision problem. Recall that M. Dehn in 1912 
[4] (see [26, Ch. 1, §2; Ch. 2, §1])) has formulated in particular 

Conjugacy problem: Does there exists an algorithm to determine whether an arbitrary 
pair of group words U, V in the generators of G define conjugate elements of G? 

The following question was posed by G. Makanin [30, Question 10.26(a)]: 
Question: Does there exists an algorithm to determine whether for an arbitrary pair 
of group words U and V oi a. free group G and an arbitrary automorphism of G the 
equation (f){X)U = VX solvable in G? 

In [2] the following problem, which generalizes the two above problems, was posed: 
Twisted conjugacy problem: Does there exists an algorithm to determine whether for 
an arbitrary pair of group words U and V in the generators of G the equality (f){X)U = VX 
holds for some W E G and (p E H, where if is a fixed subset of Aut(G)? 

In [2] a partial affirmative question to the Makanin's question is obtained. 
We will discuss the twisted conjugacy problem for H = {</)}. 

Theorem 7.1. The twisted conjugacy problem has the affirmative answer for G being 
polycyclic-by-finite group and H be equal to a unique automorphism (p. 

Proof. It follows immediately from Theorem 4.2 by the same argument as in the paper 
of Mal'cev [28] (see also [31], where the propperty of conjugacy separabihty was first 
formulated) for the (non- twisted) conjugacy problem. □ 

In fact we have proved the following statement. 

Theorem 7.2. If G is strongly twisted conjugacy separable then the twisted Dehn conju- 
gacy problem is solvable for any automorphism of G. 
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Also one can study some more particular cases of this problem. In particular, one has 

Theorem 7.3. Let G be a (p-conjugacy separable group. Then the twisted Dehn conjugacy 
problem is solvable for (p. 

The results of Section 9 imply 

Theorem 7.4. If G is a residually finite group and R{(f)) < oo, then the twisted Dehn 
conjugacy problem is solvable for (p. 

From Corollary 3.4 and Proposition 3.5 in [29] one can obtain 

Theorem 7.5. Suppose G is the fundamental group of a closed hyperbolic surface and 
(j) : G ^ G is virtually inner. Then the twisted Dehn conjugacy problem is solvable for cp. 

8. Some questions 

It is evident, that any conjugacy separable group is residually finite (because the unity 
element is an entire conjugacy class). This argument does not work for general Reide- 
meister classes. In this relation we wish to formulate several questions: 

Question 1: Does the 0-conjugacy separability imply residually finiteness? 

Question 2: Does the 0-conjugacy separability imply residually finiteness, provided 
R{(p) < oo? 

Question 3: Does the twisted conjugacy separability imply residually finiteness, pro- 
vided the existence of (p with R{(p) < ool 

Question 4: Let G be a residually finite group and cp its automorphism with R{(p) < oo. 
Is G 0-conjugacy separable ? 

The affirmative answer to the last question implies twisted Burnside-Frobenius theorem 
for (p. This will be made in Section 9. 

9. Residually finite groups 

Recall that G is called residually finite, if for any g & G there exists an (epi)morphism 
Fg of G onto a finite group Kg such that Fg{g) 7^ e G Kg. In other words, {e} G G is 
closed in the profinite topology. 

Theorem 9.1. Let G be a residually finite group and cp its automorphism with R{(p) < 00. 
Then G is (p-conjugacy separable. In particular, R{(p) = Sf{(p). 

Proof. First of all, the above semidirect product F is also residually finite [28]. Let us fix 
a couple of representatives g and h of two distinct 0-conjugacy classes in G, i.e., 

g 7^ xh(p{x~^) for any x G G. 

Since F is residually finite, we can find for each x G G an epimorphism Fr^ -.T ^ K^ such 
that 

FM) + F,{xh(P{x-^)t), FM ^ F,{xh(P{x-^)). 

We have written here the first inequality only for the purpose to keep relation to the 
previous sections. Considering the decomposition of the regular representation of K^ into 
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irreducible ones we can find a (finite-dimensional) irreducible representation of F in ^ 
with the same property 

Let px be its restriction on G <l F. Then px{g"^xh(f){x~^)) ^ 1. The representation p^ of 
G may be reducible. Let us decompose it into irreducible components: px = Pi® ■ ■ ■ ® Ps, 
V = Vi ® ■ ■ ■ ® Vs- Note that Pxif) is an intertwining operator between p^ and p^ o 0. 
This implies in particular that the kernel of p^ in C*{G) is 0-invariant. 

Now let VI := Vi, = Px{t){Vl), . . . , K' = {Px{t)Y^\Vl) . . . . Since pi is irreducible 
and Px{t) is an intertwining operator, they are spaces of (equivalent) irreducible repre- 
sentations p'l, . . . p'g, . . . of G, Pi{g) = Px(t^)pi{g)Px(t~'^)- They are linearly independent. 
Indeed, suppose V(, ■ ■ - V^ are linearly independent, but V^.^-^ fl (y( © ■ ■ • © V^) ^ 0. Since 
p'l^_^_-^ is irreducible, V^^^ C (V/©- ■ ■©V'^')- Hence, V(®- ■ -©V^' is invariant for px and Px(t). 
A contradiction with the irreducibility of px (if VI®- ■ -^V^ ^ V). Thus, V^© - ■ ■© V^' = V" 
for some k. By the definition, p[, . . . p'j^ are pairwise equivalent, in particular, they have 
the same kernels in C*{G). Hence, their sum has the same (0-invariant) kernel. Thus, 
(1) the kernel of p[ is 0-invariant, i.e., its class is a fixed point of 0; (2) the inequality 
Px{g~^xh(j){x~^)) 7^ 1 implies p[{g~^xh((){x~^) — 1) 7^ 0. Denote p'l by tt^.. By (1) we have 
TTx o ~ T^x, i-e. the class of this representation is a fixed point of (p on Gf. To complete 
the proof it remains to prove that has only finitely many such fixed points: tti, . . . , tttv- 
Indeed, in that case the finite subgroup of vri(G) © ■ ■ ■ © hn^G) being the image of the 
diagonal representation can serve as a finite group, which distinguishes {g}<j, and {h}^. 

In fact, we will prove that = 5'/(0) < R{(f))- It is sufficient to verify the following 
two statements]^ 

1) If p G Fix (f)f, there exists one and only one (up to scaling) twisted invariant functional 
on p{G*{G)) (this is a finite full matrix algebra M{d, C)). Any such functional has to be 
of the form a ^ Tr(6a), where a,b & M{d, C), b is fixed. This functional will be twisted 
invariant if and only if b is an intertwining operator between p and p o 0. In fact, "if" 
(which is evident) is sufficient for the present theorem, while the details of "only if" can 
be found in [12, 10, 11]. 

2) For different p the corresponding 0-class functions are linearly independent. Indeed, 
coefficients of finite-dimensional non-equivalent irreducible representations of G are linear 
independent by Frobenius-Schur theorem (see [3, (27.13)]). Moreover, coefficients of non- 
equivalent unitary finite-dimensional irreducible representations are orthogonal to each 
other as functions on the universal compact group associated with the initial group [5, 
16.1.3] by the Peter- Weyl theorem. Hence, their linear combinations are orthogonal to 
each other as well. □ 
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